Claude/score paths worker cpu xxdxvp - #147

score_paths was the last CPU-bound worker still offloading to a
ThreadPoolExecutor, so its four concurrent scorings ran in the same process
as the event loop the heartbeat pings from. Most of a scoring's time is the
feature build -- convert_path_to_components walks every edge of every path in
pure Python -- which holds the GIL; only the LMDB reads and the torch forward
release it. Under load the loop thread's turnaround degrades from
milliseconds to seconds, and once the heartbeat goes unrefreshed past
HEARTBEAT_TTL_SEC (15s) peers stop counting the worker as alive and can
XCLAIM its in-flight tasks out from under it. Past
worker_loop_stall_exit_sec (60s) the loop watchdog force-exits the pod. This
is the same failure arax_pathfinder hit on asyncio.to_thread and
aragorn_score / arax_rank hit before them; nothing here was different except
that it hadn't been migrated yet.

Scoring now goes through ProcessPoolManager like its siblings:

- score_paths_task is the child entrypoint. Only the response_id and the
  task's log level cross the boundary -- the message is loaded, scored and
  saved inside the child, so the payload never lands on the parent's heap
  either.
- The child attaches its own QueryLogHandler and hands the formatted records
  back with the result; the parent folds them into the task's query logger,
  so the per-query scoring lines (feature build stats, score ranges) still
  reach the query's log list rather than only container stderr.
- Per-child state (biolink Toolkit, embeddings LMDB, the MLP) is built on
  first use rather than in the pool initializer, so a bad checkpoint or an
  unreadable LMDB fails one task with a traceback instead of killing children
  at startup and leaving the pool rebuilding itself in a loop. Each child
  caps torch to one intra-op thread: the pool is already sized to the pod's
  CPU allocation, so a full thread pool per child just oversubscribes it.
- The read-only LMDB is opened with lock=False in every child, so the pages
  are shared through the page cache instead of copied per child.
- The parent validates the embeddings cache and the weights file at startup,
  before any child spawns, so a bad mount still fails fast instead of
  surfacing as every task failing one at a time.

This also brings the OOM self-heal and the pool_task_timeout_sec (300s)
per-task ceiling to this stream; scoring previously had no timeout at all, so
a pathological message could hold its slot indefinitely.

Pool size comes from resolve_pool_workers (cgroup-aware, POOL_MAX_WORKERS
overrides) and doubles as the in-flight task limit, matching arax_rank.

The dispatch-concurrency regression test now covers score_paths, and gains a
second invariant: these workers must offload through ProcessPoolManager, and
must not instantiate a ThreadPoolExecutor or use asyncio.to_thread.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XpZWD8DD34dqV3mMkJHzky

Production logs show a pod dying mid-task with no traceback: a cgroup OOM
SIGKILL, which no in-process handler can catch (a Python MemoryError would
have hit the "Error scoring paths" handler and logged). Four tasks were in
flight, three of them 387,583 analyses each.

Scoring a message in one pass held three representations of the same data at
once -- the float16 row list, np.stack's copy of it, and the float32 cast:

     analyses    rows    stack    float32     peak
      138,824    2.18     2.18       4.37     8.74 GiB
      387,583    6.10     6.10      12.20    24.40 GiB
       79,906    1.26     1.26       2.51     5.03 GiB

The last line logged was a 79,906-analysis feature build completing; the very
next statement was its np.stack(...).astype(np.float32), a 1.26 GiB stack plus
a 2.51 GiB allocation on top of three 387k row lists that had been
accumulating for 138s. Those messages then outlived their worker, got
reclaimed, killed the next pod the same way, and were finally dead-lettered by
the poison-pill breaker after three deliveries.

Scoring now runs every SCORE_CHUNK_SIZE (4096) rows and frees the chunk, so
peak is ~208 MB whatever the message size instead of scaling with it. Rows are
copied straight into a float32 batch rather than stacked as float16 and cast,
so only one array of the batch exists at a time; float16 converts exactly, so
the input matrix is unchanged. The summary log lines keep their shape, with
build and MLP time accumulated separately now that the two interleave and the
score range tracked as running aggregates rather than a per-analysis list.

Verified against the real checkpoint at 1, 100, 4095, 4096, 4097, 8192 and
10000 rows: every score is bit-identical to the one-shot path, the reported
count/min/max/mean match, and the largest float32 batch allocated for a 60k-row
message is 4096 rows (132 MiB) rather than 60000 (1.89 GiB).

Note this bounds the scoring memory, not the message itself: a 387k-analysis
TRAPI payload still has to be decoded into the child's heap. What changes is
that the multi-GiB feature arrays no longer sit on top of it, the cost is one
child's rather than the parent's, and it is returned to the OS when the child
recycles.

The chunking is pinned by a static test, since CI cannot import this module --
torch, lmdb and bmt live only in the worker image.